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Abstract 

This paper considers the problem of detecting a few signals in high- 
dimensional complex-valued Gaussian data satisfying Johnstone's (2001) 
spiked covariance model. We focus on the difficult case where signals are 
weak in the sense that the sizes of the corresponding covariance spikes are 
below the phase transition threshold studied in Baik et al (2005). We derive 
a simple analytical expression for the maximal possible asymptotic proba- 
bility of correct detection holding the asymptotic probability of false detec- 
tion fixed. To accomplish this derivation, we establish what we believe to 
be a new formula for the Harish-Chandra/Itzykson-Zuber (HCIZ) integral 
Iu(p) e tT ( AUBD ') (dU), where A has a deficient rank r < p. The formula links 
the HCIZ integral over IA (p) to an HCIZ integral over a potentially much 
smaller unitary group U (r). We show that the formula generalizes to the 
integrals over orthogonal and symplectic groups. In the most general form, 
it expresses the hypergeometric function qFq of two pxp matrix arguments 
as a repeated contour integral of the hypergeometric function o-Fq of t wo 
r x r matrix arguments. 
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1 Introduction 

Much contemporary research in statistics concerns with situations where the di- 
mensionality of data is large and comparable to the number of observations (see 
special issues of the Philosophical Transactions of the Royal Society (2009) 367 
and Annals of Statistics (2008) 36). Often, the goal is to estimate or detect a 
few signals contaminated by high-dimensional noise. One general conclusion that 
seems to emerge from this research is that, in the absence of a priori sparsity 
assumptions about signals, there is a lower limit for the signal-to-noise ratio below 
which statistical inference about the signals completely fails (Johnstone and Titter- 
ington, 2009, Nadakuditi and Edelman, 2008, Nadakuditi and Silverstein, 2010). 
This limit equals the phase transition threshold studied in Baik et al (2005). In a 
recent paper, Onatski et al (2012) show that not all is lost below the threshold. 
They consider the case of a single non-sparse signal in high-dimensional noisy data 
and establish sharp non-trivial limits for the asymptotic power, as both the data 
dimensionality and the number of observations go to infinity, of statistical tests for 
signal detection when the signal may be arbitrarily weak. 

This paper extends Onatski et al (2012) to the case of multiple non-sparse arbi- 
trarily weak signals when the data are complex-valued. Complex-valued data are 
of interest in signal processing (Schreier and Scharf, 2010), wireless communication 
(Telatar, 1999, Tulino and Verdu, 2004), and the spectral analysis of economic and 
financial time series (Onatski, 2009). Considering the case of multiple signals is 
important for applied work because the constraint that there is no more than one 
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signal can rarely be justified in practice. We derive a simple analytical expression 
for the maximal possible asymptotic probability of correct detection, based on the 
sample covariance eigenvalues of the data, holding the asymptotic probability of 
false detection fixed. 

We find that the asymptotic probability of detection may be close to one even 
in cases where the strength of all signals is substantially below the phase transition 
threshold. This finding is, perhaps, surprising in light of the fact (Peche, 2003) 
that in such cases, sometimes referred to as the sub-critical regime, the asymptotic 
behavior of any finite number of the largest sample covariance eigenvalues is not 
different from their behavior when the data are pure noise. We show that in these 
difficult cases, the detection power lies not in the different behavior of a few of the 
largest eigenvalues, but in the small deviations of the empirical distribution of all 
the eigenvalues from the Marchenko-Pastur limit (Marchenko and Pastur, 1967). 

Let us discuss our findings in more detail. We assume that data consist of n in- 
dependent observations of p-dimensional complex- valued Gaussian vectors X t with 
mean zero and covariance matrix a 2 (l v + VHV') , where I p is the p-dimensional 
identity matrix, a is a real scalar, H is an r x r real diagonal matrix with elements 
hj > along the diagonal, and V is a (p x r)-dimensional complex parameter 
normalized so that V'V = I r . Such a spiked covariance model was proposed by 
Johnstone (2001) as a simple model of a situation, often observed in applications, 
where a few eigenvalues of the sample covariance matrix, corresponding to signals, 
are relatively large, whereas the rest of the eigenvalues are relatively small and 
tightly clustered. In our notation, the size of the spikes is regulated by the values 
of hj, and the signal space is spanned by the columns of matrix V. 

Let Xi > A 2 > ... > A p be the ordered eigenvalues of XX' /n, where X = 
[Xi, X n ], and let A = (Ai, A m ) , where m = min{n,p}. We are interested 
in the asymptotic power of tests for signal detection based on the information 
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contained in A when p, n — > oo so that p/n — >■ c with < c < oo. Our null 
hypothesis is H : hi = ... = h r = (no signals), and our alternative is Hi : /ij > 
for some i = 1, r. The matrix V is left as an unspecified nuisance parameter. In 
this framework, signal detection tests can also be interpreted as tests of sphericity. 

We consider both cases of specified and unspecified a 2 . For the purpose of 
brevity, in the introduction, we will discuss only the case of specified a 2 = 1. First, 
we study the likelihood ratio L (h; A) , defined as the ratio of the densities of A cor- 
responding to unrestricted h and restricted h — 0, the densities being evaluated at 
the observed value of A. We show that L (h; A) can be represented in the form of the 
determinant of an r x r matrix with entries equal to contour integrals of elementary 
functions. We use Laplace approximations to these contour integrals to show that 
for any h such that < h < \/c, with yfc being the value of the phase transition 
threshold, the sequence of log-likelihood processes {In L (h\ A) ; h G [0, h] r } con- 
verges weakly to a Gaussian procesl] {C\(h)]h G [0, h] r } under the null hypothesis 
as n,p — > oo. The limiting process has mean E = | YHj=i m (1 — hihj/c) 

and autocovariance function Cov (^C\ (h) , C\ {h\ j = — Y^j=i m fl — hjij/cj . 
The established weak convergence of statistical experiments implies, via Le Cam's 
first lemma (see van der Vaart 1998, p. 88), that the joint distributions of the sam- 
ple covariance eigenvalues under the null and under alternatives associated with 
h G [0, y/c) r are mutually contiguous. 

An asymptotic power envelope for eigenvalue-based tests of Hq against H% can 
be constructed using the Neyman-Pearson lemma and Le Cam's third lemma. We 
show that, for tests of size a, the maximum achievable asymptotic power against 

where $ is 



a point alternative h = (hi, h r ) equals 1 — $ $ 1 (1 — a) — yW 
the standard normal distribution function and W = — Y^i 7= i m (1 — hihj/c). A 



1 Here the index A in the notation C\(h) is used to distinguish the limiting log- likelihood 
process in the case of specified a 1 — 1, from that in the case of unspecified a 2 , which we denote 
by£». 
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preliminary analysis indicates that the asymptotic power of the likelihood ratio 
test based on the information contained in A is close to the asymptotic power 
envelope. In contrast, we find that the asymptotic powers of various previously 
proposed tests are well below the envelope. 

The central technical result of this paper is the contour integral representa- 
tion of the likelihood ratio. To derive such a representation, we establish what we 
believe to be a novel formula for the hypergeometric functions of two matrix argu- 
ments o-^o (A -^0 > w here a p x p matrix A has rank r < p, so that, without loss 
of generality, only its upper-left r x r block A is non-zero. Such functions appear 
as a key term in the explicit expressions for the joint density of the eigenvalues of 
Wishart matrices with spiked covariance parameter. In Lemma 1, we show that 

Ft ] (A, B) = —1^ I ... I Ft } (A, Z) W W (A, B, Z) f[ d Zj , (1) 

where Z = diag (zi, ...,z r ) is an auxiliary matrix, and u/ Q ) (■) is a simple function 
of A,B, and Z. This formula expresses the hypergeometric function of high- 
dimensional arguments as a repeated contour integral of a hypergeometric function 
of low-dimensional arguments, which is convenient for analysis. 

For the special case r = 1, (P) reduces to the formula that has been recently 
derived in Mo (2011) and, independently, in Wang (2012) and Onatski et al (2012) 
(see also Forrester, 2011 for a short derivation). Our method of proof is different 
from the methods used by these authors. It is based on the orthogonality of Jack 
polynomials with respect to the torus scalar product (Macdonald (1995), Chapter 
VI, §10). 

Although our analysis of signal detection in complex data requires only the 
formula for o-^o^ (A -£>) , we establish (JTJ for all a = 2//3, where (3 is a positive 
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integer^ The importance of finding "serviceable approximations" to o-^o (A B) 
has been recently emphasized by Johnstone (2007, p. 322). Since in applications 
that rely on the spiked covariance matrix framework r is typically much smaller 
than p, analyzing q.F (A Z) is much easier than analyzing o-^o 2 '' (A B), and the 
established contour integral representation of the latter may be of the welcomed 
service to practitioners. 

For a = 2, 1 and 1/2, function qFq (A,B) has an integral representation 
fg (a){p) e tr ( AGBG_1 ) (dG) , where (p) is the orthogonal group 0(p) for a = 2, 
the unitary group U (p) for a = 1, and the compact symplectic group Sp (p) for a = 
1/2, and where (dG) is the normalized Haar measure over (p). Such integrals 
have various important applications in mathematics and physics, where they are 
referred to as Harish- Chandra/It zykson-Zuber (HCIZ) integrals (Zinn- Justin and 
Zuber, 2003). The HCIZ integrals with rank-deficient A have been used in the 
analysis of spin glasses (Marinari et al, 1994), wireless communication systems 
(Muller et al, 2008), statistical tests for signal detection (Bianchi et al, 2010, and 
Onatski et al, 2012), distribution of the largest sample covariance eigenvalue (Mo, 
2011, and Wang, 2012), and spiked Wishart /3-ensembles (Forrester, 2011). Their 
asymptotic behavior as p — > oo has been studied in Guionnet and Maida (2005) 
and Collins and Sniady (2007). We hope that the reduction of HCIZ integrals over 
large group Q^ a ' (p) to those over smaller group Q^ 1 ' (r) that follows from ([T]) will 
be useful in a wide spectrum of applications. 

The rest of this paper is organized as follows. In Section 2, we derive explicit 

formulae for the likelihood ratios. Section 3 establishes relationship (CQ). Section 4 

uses ((I]) to derive contour integral representations for the likelihood ratios. Section 

5 applies Laplace approximations to the contour integrals in the derived represen- 
tor cases where (3 is odd, we require that p — r + 1 be even. For even /3, such a requirement 
is not needed. 
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tation to obtain the asymptotics of the likelihood ratio process. This asymptotics 
is then used along with the Neyman- Pearson lemma and Le Cam's third lemma to 
establish a simple analytical formula for the maximal possible asymptotic probabil- 
ity of correct signal detection holding the asymptotic probability of false detection 
fixed. Section 6 concludes. All proofs are relegated to the Appendix. 

2 Likelihood ratios 

As mentioned above, we assume that data consist of n independent observations 
of p-dimensional complex- valued Gaussian vectors X t ~ iV c (0,£). This means 
that X t = Y t + iZ t , where i denotes the imaginary unit, and the joint density of 

(Y t , Z t ) at (y, z) equals j^fA^r, ex P (~ tr [ S 1 (.V + ' lz ) (v ~ ' lz )'] } ( see > for exam- 
ple, Goodman, 1963). Further, we assume that the covariance matrix E equals 
a 2 (l p + VHV') , where H = diag (hi,..., h r ) quantifies the sizes of the covariance 
spikes. Our goal is to study the asymptotic power of tests of H Q : hi = ... = h r = 
against Hi : hi > for some i — 1, r. 

If a 2 is specified, the model is invariant with respect to unitary transformations 
and the maximal invariant statistic is A, the vector of the first m = min {n, p} eigen- 
values of XX' /n, where X = [X ± , X n ]. Therefore, we consider tests based on 
A. If a 2 is unspecified, the model is invariant with respect to the unitary transfor- 
mations and multiplications by non-zero scalars, and the maximal invariant is the 
vector of normalized eigenvalues /x = /J m _i) , where /j^ = A-,/ (Ai + ... + A p ). 

Hence, we consider tests based on /i. Note that the distribution of /i does not de- 
pend on a 2 , whereas if a 2 is specified, we can always normalize A dividing it by a 2 . 
Therefore, in what follows, we will assume without loss of generality that a 2 = 1. 

Let h — (hi, h r ), and let us denote the joint density of Ai, A m as p\ (x; h) , 
x = (xi, ...,x m ) e (R + ) m and that of n x , ...,n m _i as (y; h) , y = (y x , y m -i) G 
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(H+)m-i_ We have 

Px (x; *) = 7 f;r /(i^n ^ / e— (n^-) (dG) > (2) 

U( P ) 

where 7 depends only on n and p; II = diag ((1 + h\)~ , (1 + /i r ) _1 , 1, l); 
X = diag (xi, ...,x m , 0, 0) is a (p x p) diagonal matrix, so that there are no zeros 
along the diagonal if m = p; U (p) is the set of all p x p unitary matrices; and (dG) 
is the invariant measure on the unitary group U (p) normalized to make the total 
measure unity. Formula ([2]) is a special case of the densities given in James (1964, 
p. 489) for n > p and in Ratnarajah and Vaillancourt (2005) for n < p. 

Let s = Xi + ... + x m and let yj = Xj/s. Note that the Jacobian of the coordinate 
change from (xi,...,x m ) to (yi, y m -i, s) equals s m ~ l . Changing variables in (j2J) 
and integrating s out, we obtain 

rr m Jv- n \y\ m ( v ._ v .) 2 r oo r 
pAy,h)=l [U ^ r J) I s^ 1 / e" nstr ( UGyG - 1 HdG)ds, (3) 

U( P ) 

where 3^ = diag (yi, ...,y m -\, 0, 0) is a (p x p) diagonal matrix. 

Consider the likelihood ratios: L(h;X) = p\(\;h)/p x (\;0) and L(h;fi) = 
p M (//; h) (//; 0). Formulae (J2]) and (j3J) imply the following Proposition. 

Proposition 1 Let U (p) be the set of all p x p unitary matrices. Denote by {dG) 
the invariant measure on the unitary group U (p) normalized to make the total 
measure unity. Further, let A = diag (Ai, \) and M = diag . Then 

L{h-\) = f[(l + hi)- n J e- nt < {u - I)GAG ") (dG) and (4) 
i=1 U(p) 

L (/i; H) = n[=l(l + ^)" n ^ P r s n P -l e -ns [ e -nstr( { n-I)GMG-l) {dG)ds _ {5) 

r (rap) Jo J 

«Cp) 

8 



Our analysis of the asymptotic power of tests for signal detection is based on a 
study of the asymptotic properties of the likelihood ratio processes {L (h; A) ; h G (R + Y } 
and {L (h; p) ; hE {R + )' }• First, we will focus on the key terms in the expres- 
sions (jlj) and (jSJ), which are the integrals over the unitary group. These in- 
tegrals are special cases of the complex hypergeometric function 0-^0"^ (A, B) = 
Iu(p) etr ^ AGBG (d^O > wnere A and, possibly, B are rank-deficient. In the next 
section, we derive a formula for o-^o (A, B) with rank-deficient A and B that 
links this function to a hypergeometric functions of full-rank matrix arguments 
of lower dimensions. We do not restrict attention to the case a = 1 because, as 
discussed in the introduction, other cases constitute independent interest. 

3 Contour integral representation for qFq (A B) 

Let us first provide a necessary background on hypergeometric functions. Let A 
and B be Hermitian p x p matrices over real, complex, or quaternion division 
algebra. The eigenvalues of such matrices are real and we will denote them as 
a = (ai,...,a p ) and b = (pi,...,b p ). The hypergeometric function Fq(A,B) is 
defined as (see, for example, Koev and Edelman, 2006) 

F^(AB) ^ ^^W^W (6) 
F \A,B) - — y— , (6) 

where C { K a) (A) = C { K a) (a) , C { K a) (B) = C { K a) (b) and C { K a) (I p ) = C ( K a) (1, 1) are 
normalized Jack polynomials (Macdonald, 1995, chapter VI, §10), and the inner 
sum runs over all partitions k of k, that is over all non-increasing sequences of non- 
negative integers k = (k±, K2, •••) such that k% + K2 + ... = k. The normalization of 
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,x p ) is chosen so that 



( Xl + ... + Xp ) k = J2ci a) { Xl ,...,x p ). 



(7) 



Note that o-^o (A B) depends on A and B only through a and b. Therefore, 
in what follows, without loss of generality, we will consider only diagonal matrices 
A = diag (fli, a p ) and .6 = diag (bi, ...,b p ) . We will allow aj and bj to be complex, 
thus extending definition ([6]) to complex diagonal matrices A and B. 

As was mentioned in the introduction, for a = 2, 1 and 1/2, hypergeometric 
functions o^o^ (A B) admit the integral representation 



where Q^ a ' (p) is the orthogonal group O (p) for a = 2, the unitary group U (p) for 
a = 1, and the compact symplectic group Sp (p) for a = 1/2. For real diagonal A 



Er=o£ K hfc h C * a) {AGBG- 1 ) = e tr ( AGBG which follows from (jZj). For complex 



diagonal A and S, the representation holds by the analytic continuation because 
both parts of equality (jHJ) are complex analytic functions of the diagonal elements 
of A and B. 

The main result of this section is as follows. 

Lemma 1 Let A = diag (ai, a p ) and B = diag (bi, ...,b p ), where aj and bj are 
real or complex numbers. Assume that aj ^ for 1 < j < r and aj = for 
r < j < p, and denote the upper left block of A, diag (a±, ...,a r ) , as A. Further, 
let Z =diag (zi, z r ) , where Zj are complex variables, and let K, be a contour in 
the complex plane that encircles bi,...,b p counter-clockwise. Finally, let a = 2/(3, 
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where f3 is a positive integer. Then, assuming that p — r + 1 is an even integer in 
cases where (3 is odd, and without this additional assumption in cases where (3 is 
even, we have 

F (Q) (A, B) = —I—, I ... / F (Q) (A, Z) W W (A B, Z) f[ dz v (1) 

where 

OJ^(A,B,Z) = (_l)K-D/(2a)J ! J 

3=1 

3>i 3=1 

The proposition reduces o-^o"'' (A B) > a hypergeometric function with poten- 
tially high-dimensional matrix arguments, to a repeated contour integral of o-^o (A Z) , 
a hypergeometric function with matrix arguments of possibly much lower dimen- 
sions. In the special case where r = 1, o-^o i^Z) = e aiZl and ([T]) becomes 

Ft ] (A,B) =T (p/a)al- p/a ^- (f e a ^f[( Zl -b s )-^ d Zl . (9) 

For a = 2, this formula has been established by Mo (2011), who used it to an- 
alyze the asymptotic behavior of the largest eigenvalue of a rank-one perturbation 
of a real Wishart matrix. He gives two proofs of the formula. One of the proofs 
uses Jack polynomial expansions and requires that p be an even integer (consistent 
with our requirement that p — r + 1 is even). The other proof, which Mo (2011) 
calls geometric, allows for odd p. 

Similar to the first proof of Mo, our proof of Lemma 1 uses Jack polynomial 
expansions. In contrast to that proof, we do not rely on the simplification of the 
Jack polynomials for top-order partitions, but use Jack polynomials' orthogonality 
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r((p+l-j)/a)r(l/q) 
r((r+l-j)/a) 



a 



l-(p-r+l)/Q 

j 



-1/a 



with respect to the torus scalar product (Macdonald, chapter VI, §10). It is likely 
that our requirement that p — r + 1 is even in cases where (3 = 2/ a is odd can 
be lifted without affecting relationship (CQ). This would require a different proof of 
the proposition, which is left for future research. 

For a = 2 and a = 2/(5 with even (5, formula (Q has been independently estab- 
lished by Wang (2012). He uses the formula to study the asymptotic distribution 
of the largest eigenvalue of the real, complex and quaternionic Wishart matrices 
perturbed by matrices of rank one. Wang's proof is similar to the first proof of Mo 
(2011) (see Forrester, 2011, for an alternative proof). For a = 2, formula (Q has 
also been independently established by Onatski et al (2012). Their proof is based 
on the properties of the so-called Lauricella function. 

In contrast to (Q, the general relationship ([I]) contains special functions on 
both left- and right-hand sides. However, for a = 1, it is possible to further 
simplify the right-hand side of (CQ) using Harish-Chandra/Itzykson-Zuber formula 
(see Harish-Chandra, 1957, and Itzykson and Zuber, 1980) 

where V r (A) = YYj > i( a j~ a i) an d V r (Z) = YYj>i ( 

the Vandermonde 

determinants associated with the diagonal elements ai, a r of A and the diagonal 
elements zi, z r of Z, respectively. Using (flOl) in (CQ), noting that one of the terms 
in the definition of (A, B, Z) equals V r (Z) 2 , and applying Andreief 's identity 
(Andreief, 1883) 

det ( / f i {x)g j {x)dfi(x)) =- I ... / det (Ji (%)) det (xA) T[dfi (xj) 
we obtain the following Corollary. 
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Corollary 1 Under assumptions of Lemma 1, 

An alternative way of deriving ( TTTj) is to apply l'Hopital's rule to the Harish- 
Chandra/Itzykson-Zuber determinantal formula 

^^-vS^m^y^' (12) 

the right-hand side of which is degenerate because A is rank-deficient. We include 
a proof of ( II ip that uses this approach in the Supplementary Appendix. The proof 
is elementary in the sense that it does not rely on properties of Jack polynomials. 



4 Likelihood ratios as contour integrals 

Combining Proposition 1 and Corollary 1 leads to useful contour integral repre- 
sentations of the likelihood ratios L(h;X) and L(h;fi). We now introduce new 
notation to express such representations in a convenient form. For any z G K., let 
us define a random variable 

p r 
A p (z) = J]mO- Xj) - p / In (z - X) &F P (A) , (13) 

3=1 J 

where T v (A) is the cumulative distribution function of the Marchenko-Pastur dis- 
tribution with a point mass of max (0, 1 — c~ x ) at zero, where c p = p/n, and 
density 

ip p (x) = {bp-x) (x-Op), (14) 
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where a p = (l — ^fc^f and b p = (l + ^fc^f . Further, let 

fi(z) = -(^^z-c p Jln{z-X)dT p (X)j and (15) 
9j (z) = z^expi-Apiz)}. (16) 

Finally, for any permutation p of the sequence (1,2, ...,r) and any vector z = 
(zi, ...,z r ), let 

(r , \ -p(n-r)-r{r+l)/2 ( r , *\ 

e 4§^}> (17) 

where 5 = Ai + ... + A p . 

Proposition 2 Let t/je contour K. that encircles Xi,...,X p counter-clockwise be 
chosen so that for any z £ JC, Re 2 < fX^f=i T+K^j ^ ■ Then 

L(h;X) = h det ( I e~ nMz) gj (z)dz j and (18) 

1<«J<»* \27T1 / 

L(/i;/i) = k 2 J2^v I- (f q P (z)f[{e- n M^ gj (z 3 )}dz r ...dz u (19) 
l Z7n J J/c Jk. =1 



where i denotes the imaginary unit, the summation in / fTPj) zs over a// permutations 
p of the sequence (1,2, ...,r) ; 

r r 

k X = (-Ij^/ 2 n -Pr+r(r+D/2 "Q ^. _ h .yl "Q \jr-p ^ + ^ _ ^ 

i>j t=l 

k 2 = fcj ( n 5) pr - r(r+1)/2 r (p (n - r) + r(r + l)/2) [r (np)]" 1 . 

In the next section, we perform the asymptotic analysis of L (h; A) and L (h; p) 
that relies on the Laplace approximations of the contour integrals in ( Fl8l) and ( [191) 
after the contours are suitably deformed without changing the value of the inte- 
grals. 
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Figure 1: Contour /Q (see CT-flZI])). 

5 Asymptotic analysis 

Consider contours /Q with i = 1, r which are obtained by deforming the contour 
K defined in Proposition 2 so that /Q passes through 

(1 + hi) (cp + ^) 
— ; • 

Precisely, we define ICi as /Cj = /C i+ U /Cj_, where /Q_ is the complex conjugate of 
/Q_|_ and /Cj + = ICu U /Cj2 with 



Figure [T] illustrates the choice of /Cj. 

It is possible to verify that, when < hi < y/c^, the derivative of fi (z) equals 
zero at z i0 . Therefore, choosing contours of integration so they pass through Zio 



{zio + it : < t < 3Zio} and 



(20) 



{x + 3iZiQ : — oo < x < z i0 } . 



(21) 
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allows us to use the method of steepest descent in the asymptotic analysis of the 
corresponding integrals in (fl8l) and ([19]) . The next lemma shows that the change of 
contours in ( fl8|) and ( fT9j) does not lead to a change in the value of the corresponding 
integrals. 

Lemma 2 Suppose that the null hypothesis is true, and let h be an arbitrary num- 
ber such that < h < y/c. Suppose further that hi < h for all i = 1, r. Then, 
as n,p — > oo so that c p — > c G (0, +oo) , 



e~ nh{z) g 3 (z)dz = and 



L"'L~L "I K (z) n {e- 



-™/p(j)( 2 J') 



gj (zj)} dz r ...dzi = 0. 



Our next lemma establishes Laplace approximations to the contour integrals in 
ffl8|) and ffl9l) after the change of the contours. The lemma uses some new notation 
that we introduce now. When f\ (z) is analytic at z^, let fi s with s = 0, 1, ... be 
the coefficients in the power series representation 



(22) 



s=0 



When fi (z) is not analytic at Zj , let the coefficients fi S be arbitrary numbers for 
all sGN. 

Lemma 3 Under the conditions of Lemma 2, 



- nMz) gj (z)dz = e~ nho 



9j (zm) 



71 



1/2 



+ 



and 



(23) 
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<f ... I q p (z)f[{e- n ^^g j (z j )}dz r ...dz 1 

JKf>m JKpir) j = l 

J=l J p(j)2 n 3=1 PW L 

where O p (1) is uniform in hi,...,h r G [0, 7z] , and z = (^p(i)O) ••■ ) 2; p(r)o)- The 



— 1. 



branch of the square root in formulae W3\) and (24\ ) is chosen so that (— l) 1 ^ 2 

Using Lemma 3, we establish the following theorem. 

Theorem 1 Suppose that the null hypothesis is true (h = 0). Let h be any fixed 
number such that < h < y/c and let C [0,7T] r be the space of real-valued contin- 
uous functions on [0, 7T| r equipped with the supremum norm. Then, as n,p — >■ oo 
so that p/n = c p — > c G (0, +oo) , we have 



and (25) 
(26) 



L(h;X) = exp(-£ A p (z l0 ) + \ £ In (l - MA 1 +0p Q 

I 1=1 i,j = l ^ C P / J V 77 

L(M = e J-± A, W + i £ (in (l-M) +^) -^£> [ + 

L t=l i,j'=l ^ ^ p ' p ' p j=l ) ^ ' 



where the O p (n^ 1 ) terms are uniform in h G (0, h\ r . Furthermore, lnL(/i;A) 
and InL (h; fi) , viewed as random elements of C [0,7T] r ; converge weakly to C\ (h) 
and (h) with Gaussian finite-dimensional distributions such that, for any h,h G 
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[0,h\\ 



i,j=l ^ ' 

Cov(£ A (/0,A (h)) =-^ln(l- 



E(a)) = ^(h(l-^)+^| )flB (l 

i,j=l 



Cov[C,{h),CAh 



^2 ( In ( 1 - kihj 1 + ^ 



(27) 
(28) 
(29) 
(30) 



Theorem 1 and Le Cam's first lemma (van der Vaart (1998), p. 88) imply that 
the joint distributions of Ai, A m (as well as those of ...,// m _ 1 ) under the null 
and under the alternative are mutually contiguous for any h G [0, yfcj' . Along with 
Le Cam's third lemma (van der Vaart (1998), p. 90), this can be used to study the 
"local" powers of tests detecting signals in noise. 

Let f3 x (h) and /3 (h) be the asymptotic powers of the asymptotically most 
powerful A- and /x-based tests of size a of the null h — against a point alternative 
h = (hi, h r ) with hj < y/c, j = 1, ...,r. We have 

Theorem 2 Let <3> denote the standard normal distribution function. Then, 



0x(h) = 1-$ 



1 - $ 




and 



( In ( 1 - hihj } + kihj 



(31) 
(32) 



The theorem implies in particular that detection of signals corresponding to 
covariance spikes of sizes well below the phase transition threshold is possible with 
high probability. Consider for example the case where the number of observations 
equals the dimensionality of data so that c = 1, the number of signals under the 
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alternative equals five, and the signals have equal but rather weak strengths h\ = 
... = h 5 = 0.5. Then the best possible A-based procedure for detecting such signals 
with the asymptotic probability of false detection fixed at 0.05 has asymptotic 



probability of correct detection 1 — $ 



0.85. 



(0.95) - ^-25 In (1 -0.25) 
Unfortunately, constructing testing procedures with uniformly optimal power 
is hard because the log-likelihood process established in Theorem 1 is not of the 
Gaussian shift type, so that the statistical experiments we study are not locally 
asymptotically normal (LAN) ones. For the case of real- valued data and r = 1, 
Onatski et al (2012) use numerical simulations to show that the asymptotic powers 
of the likelihood ratio (LR) tests based on A and on fi are close to the respective 
asymptotic power envelopes (3 X (h) and /3„ (h). The A- and /z-based LR tests of 
h = against the alternative h e (0, h,y reject the null if and only if , respectively, 
2 sup hg ( ^ In L (h; A) and 2 sup hg( - ^ InL (h; n) are sufficiently large. As r grows, 
it becomes increasingly difficult to find the asymptotic critical values for the LR 
tests by simulation. This requires simulating an r-dimensional Gaussian random 
field with the covariance function and the mean function described in Theorem 1, 
which, for relatively large r, is computationally expensive. 

For r = 2, Figure [2] shows the contour plots of the power envelope f3 x (h) (left 
panel) and of the asymptotic power of the likelihood ratio test based on A. We chose 



parameter h so that it is very close to the threshold a/c, precisely h = a/c(1 — e~ 36 ). 
We see that the contours of /3 A (h) and of the asymptotic power of the A-based LR 
test corresponding to the same value of these functions are relatively close to each 
other, which suggests that the LR test has good asymptotic power properties. 
More detailed analysis of the asymptotic and finite sample power of the LR test 
is, however, beyond the scope of this paper, and is left for future research. 

In contrast to the LR test, the popular signal detection procedures based on 
the information in a few of the largest eigenvalues of XX' /n (see, for example, 
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Power envelope Power of the LR test 




h/o" 2 h/c" 2 

Figure 2: The asymptotic power envelope f3 x (h) and the asymptotic power of the 
LR test based on A; r = 2, asymptotic size is 0.05. 

Krichman and Nadler (2009), Nadakuditi and Silverstein (2010), Onatski (2009), 
Patterson et al (2006), Perry and Wolf (2010), and Tracy and Widom (2009)), 
have trivial asymptotic power (that is, the asymptotic power, which equals the 
asymptotic size) in the region h G [0, y/c) ■ It is because the asymptotic behavior 
of any finite number of the largest sample covariance eigenvalues when h G [0, y/c)' 
is not different from their behavior when the data are pure noise (Peche, 2003). 

As was mentioned above, signal detection tests can be interpreted as tests of 
sphericity. Vice versa, previously proposed sphericity tests, can, in principle, be 
used for signal detection. In the Supplementary Appendix, we use Theorem 1 
along with Le Cam's third lemma to derive asymptotic powers of several such 
tests against "spiked covariance" alternatives. The derived asymptotic powers 
turn out to be much lower than the asymptotic power envelopes /3 A (h) and (3 (h). 
However, we feel that this comparison is somewhat unfair to the sphericity tests 
because they are typically designed against general alternatives, as opposed to "the 
spiked covariance" alternatives. Therefore, and to save space, we do not report 
these results here. 
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6 Conclusion 



This paper studies the asymptotic power of the signal detection tests in complex- 
valued Gaussian data as both the number of observations and data dimensionality 
go to infinity. Contrary to the conventional wisdom that detection of signals be- 
comes nearly impossible when their strength, measured by the size of the covariance 
spikes, is below the phase transition threshold, we find that detection of such sig- 
nals may be possible with high probability. The detection power lies not in the 
different behavior of a few of the largest sample covariance eigenvalues under the 
null and the alternative, which is exploited by the popular signal detection tests, 
but in small deviations of the empirical distribution of all the eigenvalues from the 
Marchenko-Pastur limit. 

To derive our results, we consider the ratio of the densities of the sample co- 
variance eigenvalues under the null and under the alternative hypothesis. We 
establish a contour integral representation of this likelihood ratio, and use the 
Laplace approximation to derive its asymptotic limit. Our analysis of the limiting 
log-likelihood ratio process shows that the sub-critical region, where the sizes of the 
covariance spikes are below the phase transition threshold, is the region of mutual 
contiguity of the joint densities of the sample covariance eigenvalues under the null 
and the alternative. We use the derived limiting log-likelihood process along with 
Le Cam's third lemma and the Neyman-Pearson lemma to obtain the asymptotic 
power envelope for the signal detection tests. Preliminary analysis indicates that 
the asymptotic power of the likelihood ratio test based on the sample covariance 
eigenvalues is close to the asymptotic power envelope. 

Our technical analysis is based on what we believe to be a novel representation 
of the Harish-Chandra/Itzykson-Zuber integral with one of the pxp matrices being 
of reduced rank r in the form of an r x r matrix of contour integrals. We obtain 
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such a representation as a corollary to a much more general result established in 
Lemma 1. This result expresses the hypergeometric function 0-^0°^ of two p x p 
matrix arguments, one of which has rank repeated contour integral of the 

hypergeometric function qF^oI two r x r matrix arguments. As discussed in the 
introduction, the established dimension reduction for the hypergeometric function 
may be important in various applied and theoretical fields of study. In particular, 
for a = 2, it can, potentially, be used to extend the analysis of this paper to the 
case of real-valued data. Such an extension is currently under investigation. 



7 Appendix 

Proof of Lemma 1. 

Let f{Z) and g{Z) be functions defined on the r-dimensional torus {\zj\ = 1, for 
j = 1, r} . Consider the scalar product, sometimes called the torus scalar prod- 
uct, 

(/ - g) ° = rthf /■■■//(2)p)i]( i -^ i )"°np < 33 > 

where the contours of integration are the unit circles in the complex plane. Our 
proof relies on the orthogonality property of Jack polynomials: (cl?\ Ci a ^ = 
for k/t (Macdonald, chapter VI, §10). 

Let us, first, introduce a few definitions (following Macdonald, 1995, chapter I, 
§1, and Dumitriu et al, 2007): The non-zero Kj in the partition n = [ki, K2, ■■■] are 
called the parts of k. The number of parts is the length of n, denoted as / (k) . The 
sum of the parts is the weight of k, denoted as \k\. We will identify partition k with 
its Ferrers diagram, defined as an arrangement of \k\ boxes in I (n) left-justified 
rows, the number of boxes in row i being the same as Ki (see Figure [3]). For each 
square s in the Ferrers diagram, let /' (s) ,l(s),a(s), and a'(s) be respectively the 
numbers of squares in the diagram to the north, south, east, and west of the square 
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k=[4,3,3,1,1], 

I(k)=5, 

|k|=12, 

a(s)=2, 

l(s)=3, 

a'(s)=0, 

l'(s)=1. 



Figure 3: The Ferrers diagram of partition [4, 3, 3, 1, 1]. 

s. Further, let h* (s) = I (s) + a (1 + a(s)) and (s) = I (s) + 1 + cra(s). Finally, 
let c {k, a) = IIsgk ^* c ' ( K ' a ) = ri seK h* (s), and w; (k, a) = c (k, a) c' (k, a). 
We will need the following lemmata. 

Lemma Al. For the torus scalar product of C« with itself, we have 

^ /a Vr(i)r(l+^)J ll r +(a'(«)+l)a-Z'( a )-l- 

(34) 

PROOF: Macdonald (1995, Chapter VI, §10) establishes the orthogonality of "P" 
normalizations of Jack polynomials, Pk°\ with respect to the torus scalar product. 



His formula (10.37) gives an explicit expression (up to a constant that can be 

:(ft,a) 



evaluated using (10.38)) for (Pi a) ,Qi a) ) , where Q { " ] = ^0M a) (see (10.16)). 



On the other hand, 



\k\\ 



(see, for example, Table 6 of Dumitriu et al, 2007). Substituting this expression in 
Macdonald's formulae, we get (l34l) .D 
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Lemma A2. Let Z = diag (z\, z r ) , where Zi,...,z r are complex variables, 
and let bi,...,b p be complex constants. Then 

r p oo /\ 

II II <i-*>*r 1/a = E E (<*) • oe) 

3=1 s=l fc=0 /chfc l a 

T/ie series on the right-hand side of this equality converges uniformly over Q p = 
{Z : m&Xj< r \zj\ < p -1 } , for any p > max s < p |6 S |. 

Proof: Macdonald (1995, Chapter VI, §10) shows that f\ f[ {l-b s ZjY l/a = 

j=l 8=1 

Er=o (B) Qi a) (Z) , where = ^P^> '. This result together with 

(|35|) imply (1361) . The uniform convergence in (1361) follows from the fact that func- 
tion rij = i ris=i (1 — bgZj)" 1 ^ is analytic in an open region that includes Q p .H 

We are now ready to prove Lemma 1. Consider the right-hand side of (JTJ, which 
we will denote as RHS. We will assume that max s < p |6 S | < 1 and that the contour 
K, is the unit circle in the complex plane. That these assumptions are without loss 
of generality follows from the fact that the value of RHS does not change under the 
transformation Z — > (pZ, B — > tpB, and A — > ^ X A, where (p is any positive num- 
ber, and under a deformation of K, into the unit circle (because such a deformation 
leaves the contour in the region of the analyticity of the integrand). With these as- 
sumptions, and noting that the component Yl]>i ( z j~ z i) 2 ^ a °f u ^ (A B, Z) equals 
(-^^-^^n^i^^n^ill-^^ 1 ) 17 ", we can rewrite RHS for a = 2/(3, 
where is any positive integer, in the form of the torus scalar product 

RHS = ^ ( Ft\A,2) ^(^/aj^^flflil-b^)- 1 ^ 

j=l j=l s=l 



where 



7 w = n 

j=l <- 



T(ip+l-j)/a)T(l/a) 
T((r+l-j)/a) 



(37) 
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Substituting Fq (A, Z) and Yl Y[ P s =i (1 — b s Zj)~ ' a in the above formula by 
their expansions and ( 1361) in the series of Jack polynomials, and interchang- 
ing the order of integration and summation, which is possible because the series 
converge uniformly over the unit torus, we obtain 

rhs = >)frwlM^^) 



k=0 Khk 



But ]\ r j=l (z j ) { ^ r+l)/a - 1 (Z) = C^ a) (Z), where f denotes partition 
[ti + ...,r r + Note that f is well defined for a = 2/(3, where (5 

is an even integer. If /3 is an odd integer, we need to assume that p — r + 1 is 
even. Therefore, using the orthogonality of the Jack polynomials with respect to 
the torus scalar product, we have 



RHS = 7 («> JJ a -(— >/« £ £ ^1^(8) ( C l»)p) ,cf >(2) 



Using Lemma Al, (J37J), and equality U aj^^ * 1 ^ (A) = # (A) = (A) 



3 

i=i 



we get after some cancellations 

oc 



t=0 tH ° T l J PJ 



where 



*(«) _ aj^wjr^Cg Cg A T((p+l-j)/a) n r+g' (s) a-/' (s 

11 r (-\±( r -n\ ln,\ 11 



7 



(f,a) a 2 *t! C| a) (J r )y r(l+(r-j)/a)^r+(a'(s)+l)a-Z'(s)-l" 
In the above expression for 7^, substitute # (J p ) and C| a) (J P ) by their 
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explicit forms, that can be obtained from a general formula 

Cl a) (Im) = -j 1 -^ H(m + aa' (s) - l'(s)) . (39) 
y ' ' sen 

A variant of this formula, that uses the generalized Pochhammer symbol, can be 
found, for example, in Dumitriu et al (2007, Table 5). Then, after cancellations, 
we get 

~c«) = ^ IT r((p+i-j)/cQ IL eT (?+<*»' (s)-z'OQ) 

7 " of }ir(l+(r-i)/a)n 86f (r+(a'( S )+l)a-f(fl)' 

Now consider the last ratio of the products in the above expression. For the 
product term in the numerator that corresponds to square s in the position (i, j) 
in the diagram of r, there exists exactly the same term in the denominator, which 
corresponds to square s in the position (i,j + {jp — r + 1) /a — 1) in the diagram 
of f. Therefore, we can write 

*(«) = —Y\ r((p+i-j)/cQ l 

7 a* f = \T (lHr-j)MUseAr + (a' (s) + l)a-l' (s)-lY 

where f is the partition that consists of r identical parts (p — r + 1) /a — 1. 
Finally, note that 

J] (r + W + <*-/' = a |fH J] ((r-Z'(s)- 1) /a+a' (s) + l) 

s£t A'fcf 

,f h TT r((r-j)/a + (p-r + l)/a) = |f ,_ t ^ r ((p-j + 1) /a) 
" fi r((r-j)/a + l) " ^((r-ij/a+l)' 

Therefore, 7^ = 1 and the statement of the lemma follows from ([6]) and ( |38i) .D 
Proof of Proposition 2 

Proposition 1 and Corollary 1 directly imply (ITS]) and the following formula for 
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L(h;n) 



^npqpi — r(l+r)/2 roo 
L (h- n) = h / y P(n-r)+r(r+l)/2-l e -ny det Rdy ^ (4Q) 

1 {npj J 



where R is an r x r matrix with 



Rij = 2^ £ v_i no*- ^r 1 



Let us write det i? as 



i /• a nh P Q) p 
det J R = ^sgnpJJ — i e^+^V- 1 J]> - A,) -1 dz, 

P 3=1 m JK s=1 

or equivalently as 

r r r ( y nh pU) p "1 

det r = e £ - £ n i^^v 1 n & - ^r 1 } dz r...d Z1 . (4i) 

Using this representation, we have 

sgn p 



poo 

/ y p(n-r)+r(r+l)/2-l e -ny ^ R ^ y = V- 
^0 



X 



(2*1)' 

yP{ n-r )+ r { r + l)/2-l eX p <j - ( n - £ ) y U 



3=1 



r I v 



n\ z 3 i n^' - ^ 1 \ dzr.-.dz^y. 

3=1 I s=l 



h. 



S, the 



Since the contour /C is chosen so that for any z G /C, Re z < ^X)^=i !+/, 
integrand in the above multiple integral is absolutely integrable on [0, oo) x /C x 
... x JC, and Fubini's theorem justifies the interchange of the order of the integrals, 
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so that 



^ (n _ r)+r(7 . + l)/2 _ le _ n2/ ^ R ( y } dy 



sgnp 

(2vri) r Jk: Jk 



h 



F(p(n-r) +r(r + l)/2) 

n p(n-r)+r(r+l)/2 
-p(n—r)—r (r+l)/2 



X 



'P(j') £j 



x 



dz r ...dz\. 



8=1 



Combining this with ( 140)) . we get f fT9|) . □ 
Proof of Lemma 2 

The lemma can be proven using arguments very similar to those in the proof 
of Lemmas 4 and 6 in Onatski, Moreira and Hallin (2012) (OMH in what follows), 
and we omit the proof to save space. □ 

Proof of Lemma 3. 

To save space, we will only establish (|23|) . relegating a conceptually similar but 
more technical proof of (l24j) to the Supplementary Appendix. Lemma 5 in OMH 
implies that 



- nfdz) g{z) dz = e- n}M 



gMn 1 ' 2 | Q P (l) 



(42) 



where g (z) = exp { — |A P (z)} and O p (l) is uniform in hi G (0,/i]. A careful 
inspection of OMH's proof of their Lemma 5 reveals that a version of Q4"2l remains 
valid for general functions g (z) that are analytic in the open ball B (z i0 , r») with 
center at Zio and radius r, = min jzjo — max |6 P , Ai} , ^-j^S — with probabil- 
ity approaching 1 as n,p — > 00. Precisely, for such general g (z) we have 



e -"M*)g( Z ] 



dz = e~^ 9 ^ ^ + ^ + * 2 + tt £ 



(43) 
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with 



|^i| < Cie-^/i^TT^sup^Cx 



zeB 



|* 2 | < C x e- n ^e- nC2 h- x sup and 



|*a| < Ci 



" n/ ' (2) g(2)dz 



/Q2U/Q2 



(44) 
(45) 

(46) 



where Ci and C2 are some positive constants, and 5 is a closed ball with center 
at Zio and radius rj/2. 

Now, let g (2) = ^(-z) = z^ 1 exp {— A p (2)}. Lemma A2 in OMH implies that 
suPsefluJCuuea If (*)l = ^"^p W uniformly in ^ G (0, fe]. Therefore, by flS) and 
ED, 

*j + m 2 = e- nfM h^ n -^ 2 O p (1) . (47) 



Turning to the analysis of ^3, note that by definition of fa (z) and g (z) 



'Uiz-X^ 



(48) 



For z G /C i2 U /Cj2, we have |(z — Aj) 1 | < (3z i0 ) 1 , and \z(z — Xj) 1 | < 2, for any 
j = 1, ...,p. Therefore, using (j4"8~l) . we get 



- nh{z) g(z) dz 



Ki2UKi2 



< 2^ 1 (3z i0 ) 



-p+j-i 



2 j (3z,-o) 



-p+i-i 



hi 



n- 



1 + hi 



e dz 



2 j (3^ 



2 J (3z i0 



1J-1 



n 



n- 



-n (c p ln(3^io ) - jq^r *io) 



g-p e -n(c p \n(z i0 )-hi-c p ) 



On the other hand, for any hi G [0, /i] , /ij < y/Cp" for sufficiently large n and p, and 
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Cp In (zio) — hi — c p > /j . Indeed, using the definition of z i0 and the fact, established 
in OMH's Lemma 11, that f i0 = —c p — (1 — c p ) In (1 + hi) + c p ln ^, we have 

c p In (z i0 ) -K- Cp - f i0 = In (1 + K) + c p In (c p + /ij) - h { - c p In c p . 

The right hand side of this equality equals at hi = and has a non-negative 
derivative with respect to hi for all < hi < y /c p '. Therefore, 



- n ^ z) g(z) dz 



/Q2U/Q: 



< V (3^o 



n- 



-1 



3 -p e -n/»0 



and thus, \l/3 = e n ^ i0 hi J n 3 ' 2 O p (l), uniformly in hi G (0, /il. Combining this 



with (HI and (i7D, we obtain (1231). □ 



Proof of Theorem 1 

Proposition 2 and Lemma 3 imply that 



L A ) = fei exp{-nX)[ = i/»o} det I zio 1 exp{-A p (z i0 )} + O p (1) 



(2i) r (vm) r/2 



,1/2 

Ji2 



ten 



l<i,j<r 



As is shown in OMH (see their Lemma 11 and (A8))3, for hi < h, 



/i0 
fi2 



-c p — (1 — c p ) In (1 + /ij) + Cp In and 



/? 2 



2(1 + ^(^-/11) 



(49) 
(50) 



Moreover, by OMH's Lemma A2, exp {— A p (zjo)} — O p (l) uniformly in h G 
(0,/i] r . Using these facts and the definition of ki given in Proposition 2, we get 
after some algebra 



3 Note that the expressions given in OMH are half times the expressions given below because 
the equivalent of fi in the real- valued data case considered by OMH is fi/2. 
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L(h;X) = //'' -\\ 



t=i 



c -h 2 ^ 1/2 ' 



e r Pp -rp x 



1=1 



Applying Stirling's formula 



(p - *)! = e-*f- t+1 {^j (1 + (p- 1 )) 



we get 



L(M)=exp -^A p y m i-- n 

I i=l J t=l v i>j 



Urn 



which implies (1251) . 

Turning to the proof of ( 1261) . Proposition 2 and Lemma 3 imply that 



L(h;(j) = (-1) 



r(r-l)/2 ^- pr +r(r+l)/2 



(51) 



i>j 



t=l 



X\[h r r p {i + h t r n - i { P -ty] (ns) 

sgn 
(2^i 



-r(r+i)/2 r(p(ra-r) + r(r + l)/2) 



•'pO') 2 



Using the definition of g p (z ) and of z i0 , we get 



\ -p{n-r)-r(r+l)/2 ( r u 

- I'-Ettet H"E£x 



i=l 



i=l 



j 4" <",, 



-p(n— r) — r(r+l)/2 



i=l 



s 



exp < — pr — n/i, > . (52) 



i=i 



31 



Further, using the definition of gj {z P (j)o) , the fact that J2 P s S n P^p(j)o ec L ua l s the 
Vandermonde determinant Yll>j ( z io ~ z jo) > we § e t 

r r 

n ^ 1/2 n ^° ~ • 

i=l i>j 

Substituting fl52l) and fl53l) into fl5Tj) . and using fj49l) and (j5Dj) together with the 

— 1/2 / 

fact that the branch of the square root in / i2 is chosen so that v — 1 = — i, we 
get after some algebra 



mm = n[(p-t)!(c P -^) i/2 ]^-^ +i )/ 2 x 



t=l 

-p(n— r)— r(r+l)/2 



r (p (n - r) + r(r + l)/2) f ^ _ y> /ij- + c. 



r(np) \ ^ 5 



x 



exp \ - s ^ r ^ \r/ 2 c / r n ~ ( x + °p ( n x )) • 

I i=i J (2vrn) 4>J 

Now, using the fact that S - p = O p (1) , we get In (|J = + O p (p~ 2 ) and 

[ S V 2 



— — -5 (5-p) + O p (p ) 



p2 



Further, the Stirling approximations give 

1/2 



(p-t)\ = e 



PpP-t+^f^j (i + o^- 1 )) and 
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So finally, after some cancellations, 



-(E--i^) 2 -5^i(s- p ) -EApW 



L(/i;/l) = e a«*^=i'^ e 

1/2 r 



t=i ^ / i>j 

which implies ( 1261) . 

To establish the rest of the statements of Theorem 1 we will need the following 
lemma. 

Lemma A3. Suppose that our null hypothesis holds. Denote Y^j=i as T- 
Then, for any fixed r and h < y/c, and any (hi, h r ) e (0, li\ r , as n,p -)■ oo so 
that p/n — > c, the vector (S — p, T — (1 + c p )p,A p (z w ) , A p (z r o)) converges in 
distribution to a Gaussian vector (rj, £ 1( £ r ) wift 

Efa) = E(C) = E(e i ) = 0, 

Var(r?) = c, Var (() = 2c (2 + 5c + 2c 2 ) , Cov (77, C) = 2c (1 + c) , 

Covfaft) = -^,Cov(C,^) = -^(^ + 2 + 2c), and 

Cov (&,£*.) = -ln(l -hih k /c) 



Proof: The proof of the lemma is similar to that of Lemma 12 in OMH. 
The convergence to the Gaussian distribution follows from Theorem 1.1 of Bai and 
Silverstein (2004). The formulas for the means, variances and covariances of r] and 
£j are obtained using Theorem 1.1 iii) of Bai and Silverstein (2004) similarly to 
how the corresponding formulas in Lemma 12 of OMH are obtained using Theorem 
1.1 ii). Therefore, below we only derive the formulae for the mean, variance, and 
covariances that involve (. Variable £ does not appear in Lemma 12 of OMH 
because the lemma does not study the asymptotics of T — (1 + c p )p. 

The fact that E( = follows directly from Theorem 1.1 iii) of Bai and Silver- 
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stein (2004). The same theorem implies that 



Cov (£ ,g = --j-j * * - — — 7770 — j j dzid^, 54 

J 4tH / / (m(zi)-m(^)) cbi 

where z j0 = limz.,0 as n,p — > 00, 

/ an 1 II z 2 z i dm(z 1 )dm(z 2 ) , 

Cov 77,0 = -7-2 * : : , , „ 2 a a - dzidz 2 , 55 

47r 2 / J (m(zi) -m(z 2 )) & z x dz 2 



and 



where 



Var (0 = -7-2 f 9 / 77^^772 7 7 M zidza, 56 
47r 2 / y (m (21) -m^)) dzi dz 2 



m(z)= — (l — c)z 1 + cm(z) 
with m (2) given by (3.6) of OMH, where c p is replaced by c. That is, 



-z + c - 1 + J (z - c - i f - 4c 
m (z) = ^ , (57) 



where the branch of the square root is chosen so that the real and the imaginary 



parts of y [z — c — l) 2 — 4c have the same signs as the real and the imaginary 
parts of z — c — 1, respectively. The contours of integration in (154"]) - ( 156]) are closed, 
oriented counterclockwise, enclose zero and the support of the Marchenko-Pastur 
distribution with parameter c, and do not enclose Zjo- 

The above expressions can be simplified. Use formula 1.16 of Bai and Silverstein 
(2004), to get 

^^.^Mfcz^J^^ (58) 
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where 



Cov ( V , C) = - JL / / Z(mi)(z(m f d ffll d m2 , and (59) 
4^ J J (mi - m 2 ) 

Var {0 = ~U ^K)f(^W) 2 dmidm2| (B0) 
47r 2 J J (mi - m 2 ) 



z( m ) = -I + — J— (61) 
m 1 + m 



and the contours of integration over mi and m 2 in ( 15811601 are obtained from the 
contours of integration over z\ and Z2 in ( )54ll56l) by transformation m (z) . Recall 
that by assumption the contours over Z\ and z 2 intersect the real line to the left of 
zero and in between the upper boundary of the support of the Marchenko-Pastur 
distribution, (1 + y/c) 2 , and Zj . Therefore, as can be shown using the definition 
f )57|) of m(z), the mi-contour and m.2-contour are clockwise oriented and intersect 
the real line in between — (1 + y/c) 1 and m (zjo) = —hj (hj + c)" 1 and to the right 
of zero. In particular, both contours enclose and —hj (hj + c)~ , but not —1 and 
-(l + hj)' 1 . 

Assuming without loss of generality that mi-contour lies inside the m.2-contour, 
from (A64) in the Supplementary appendix of OMH, we have 



ln( %0 -,(mi)) dmi = 27r . + 1 (62) 

(mi - m 2 ) V ^ m 2 + hj (hj + c) 



Denoting —hj (hj + c) 1 as Xj, we get from ( 1621) and (J58 



2 



Covfc,C) = f^/w™ 2 )) 2 (- — )dm 

J An 2 J \m 2 m 2 — Xj 1 



An 2 J \ m 2 1 + m 2 / \ m 2 m 2 — Xj 
-hj (hj + 2 + 2c) , 



where the last equality follows from Cauchy's residue theorem and the fact that 



35 



the contour is oriented clock-wise. 
For Cov (rj, () , we have 



f 



(mi — m 2 ) J (mi — m% 



f z (mi) / I m i ^ !+ m i ) 27ri 



2 dmi = (p — ^-dnii = — ^ 



so that 



27ri / I (z(m 2 )) 2 



CovfoC) = -—jj—-^dm 2 

2m f f 1 c \ 2 1 , 

h — -dm 



2c(l + c) 



'2 



by Cauchy's theorem. 
For Var (() , we have 



,2 



1 



z [nil , / V ™i 1+m i / , 

jdmx = a) — 2 — dmi 

(nil — m 2) J ( m i — m 2) 

4vri / 1 
= — 2 c 



so that 



•4^ f f ^ ^ 



= 2c (2 + 5c + 2c 2 ) 

by Cauchy's theorem. □ 

Lemma A3 and formulae (J25l) and ( l26l) imply the convergence of finite dimensional- 
distributions of the random fields InL (h; A) and InL (h; u) to the Gaussian distri- 
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butions with means and covariance matrices characterized by (l27H30p . 

To complete the proof of Theorem 1, we need to establish the tightness of 
In L(/i; A) and lnL(/i;/i), viewed as random elements of the space C [0,7T] r , as 
n,p — > oo so that p/n — > c. Formulae f )25ll26|) and the facts that S — p — O p (1) , 
and that A p (z i0 ) = O p (1) for i = l,...,r, where O p (1) are uniform in h e 
(0, hY , imply that for an arbitrarily small positive e, there must exist B > such 
that Pr ^sup hg ^ |ln L(h; A) | > -B^ < e and Pr ^sup^Q ^ |lnL(/i;//)| > < 
£ for sufficiently large n and p. Since, as implied by Proposition 1, In L(h;X) 
and In L(h;fi) are continuous functions on h e [0,7T] r , sup^g^^jr |lnL (/;,; A)| = 

su P/i G [o,/i] r l ln L fa A ) I ' and su Phe(o,fe] r l ln L C 1 ! ^) I = su Pfe G [o,h] r l ln L fa I ' so that 
the tightness of InL (h; A) and InL (/i; /x) follows. □ 

Proof of theorem 2 

To save space, we only derive the asymptotic power envelope for the relatively 
more difficult case of real- valued data and /i-based tests. According to the Neyman- 
Pearson lemma, the most powerful test of the null h = against a point alternative 
h = (hi, h r ) is the test which rejects the null when InL (h; /i) is larger than a 
critical value C. It follows from Theorem 1 that, for such a test to have asymptotic 
size a, C must be 

C = (h)®- 1 (l-a) + m(h), (63) 

where 



m (h) = 
W(h) = 




Now, according to Le Cam's third lemma and Theorem 1, under h = (hi, h r ) , 
InL (h; yi) -A N (m (h) + W (h) , W (h)) . Therefore, the asymptotic power (3 (h) 
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is (|32|).D 
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